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SUMMARY:  This  technical  note  is  designed  to  help  ecosystem  restoration  planners  and  project 
managers  identify  the  best  metrics  to  evaluate  and  select  the  recommended  restoration  plan, 
monitor  and  assess  progress  toward  achieving  project  objectives,  and,  if  necessary,  inform 
adaptive  management  decisions.  Performance  metrics,  or  measurable  system  components  used  to 
estimate  and  track  the  state  of  critical  aspects  of  the  project,  are  often  the  basis  for  project 
decision  making  and  furthering  scientific  understanding.  As  such,  ecosystem  restoration  planners 
should  take  time  to  carefully  select  an  appropriate  and  effective  metric  set.  To  help  planners  with 
this  task,  this  technical  note  accomplishes  the  following: 

1.  Reviews  current  USACE  ecosystem  restoration  planning  and  monitoring  policy,  regulations 
and  guidelines. 

2.  Explains  the  importance  of  metric  selection  and  its  roles  during  planning  and  post-construction 
monitoring  and  assessment. 

3.  Reviews  common  options  for  identifying  and  selecting  metrics  including  conceptual 
modeling,  historical  precedence,  and  best  professional  judgment. 

4.  Presents  two  metric  evaluation  methods,  screening  and  multi-criteria  decision  analysis. 

5.  Discusses  metric  application  to  ecosystem  restoration  project  planning  and  monitoring. 

This  technical  note  is  expected  to  assist  ecosystem  restoration  practitioners  in  selecting  and 
applying  metrics  that  can  be  used  to  further  scientific  understanding,  resolve  hypotheses,  and 
evaluate  expected  and/or  realized  project  impacts. 

BACKGROUND:  Environmental  protection  has  been  a  primary  mission  of  the  US  Army  Corps 
of  Engineers  (USACE)  since  the  passing  of  Public  Law  101-640  in  November  of  1990.  Since  then, 
a  substantial  number  of  USACE  offices  have  been  involved  in  the  planning,  design,  and 
construction  of  water-related  restoration  projects  with  a  focus  on  wetlands,  submerged  aquatic 
vegetation,  oyster  reefs,  riparian  forest,  and  wet  prairie  (Miner  2005).  The  objective  of  these  Corps 
ecosystem  restoration  projects  is  to  increase  the  net  quantity  and/or  quality  of  desired  ecosystem 
resources  by  restoring  degraded  ecosystem  function,  structure,  and  processes  (Engineer  Regulation 
(ER)  1105-2-100  (USACE  2000)).  Planning  an  ecosystem  restoration  project  includes  identifying 
problems  and  opportunities,  inventorying  and  forecasting  conditions,  and  fonnulating  and 
evaluating  alternative  plans.  Plans  are  evaluated  based  on  an  objective  characterization  of  the 
plan's  inputs  and  outputs  (i.e.,  what  conditions  are  produced  by  a  project,  as  opposed  to  how  does 
the  presence  of  the  project  alter  conditions  that  might  have  otherwise  existed).  The  results  of  the 
evaluations  are  then  compared  against  one  another.  These  planning  steps  require  metrics  that 
indicate  the  state  of  the  ecosystem  and  clearly  identify  quantifiable  ecosystem  restoration  outputs 
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(ER  1105-2-100  (USAGE  2000)).  The  same  metrics  should  be  used  during  development  of 
monitoring  plans  to  help  decision  makers  understand  the  degree  to  which  the  project  is  delivering 
expected  results  and  to  inform  adaptive  management  decisions.  Section  2039  of  the  Water 
Resources  Development  Act  (WRDA)  of  2007  (US  Congress  2007)  recommends  that  US  ACE 
ecosystem  restoration  feasibility  studies  identify  success  criteria.  The  law  allows  for  up  to  10  years 
of  federally  cost-shared  monitoring  in  the  event  that  success  criteria  are  not  met  in  less  than 
10  years.  USACE  Implementation  Guidelines  for  Section  2039  of  WRDA  2007  specify  that  each 
ecosystem  restoration  monitoring  plan  must  include  key  project-specific  parameters  to  be 
measured  and  describe  how  those  parameters,  or  metrics,  relate  to  achievement  of  desired  project 
outcomes  or  implementation  of  contingency  options  identified  in  the  project’s  adaptive 
management  plan  (CECW-PB  (USACE  2009)).  Beyond  policy  requirements,  effective  monitoring 
is  an  essential  aspect  of  ecosystem  restoration  as  it  demonstrates  progress  and  the  degree  to  which 
objectives  are  being  satisfied  to  leadership,  stakeholders,  and  future  project  sponsors.  It  also 
increases  the  depth  and  breadth  of  understanding  about  the  effectiveness  of  ecosystem  restoration 
practices,  contributes  to  expanding  knowledge  about  ecosystems,  and  guides  management 
decisions  on  the  most  effective,  efficient,  and  economical  courses  of  action  (Thom  and  Wellman 
1996,  Grootjans  et  al.  2002;  USACE  2009).  Appropriate,  clearly  defined  metrics  are  essential  to 
successful  ecosystem  restoration  planning  and  monitoring  programs  and  must  be  identified, 
quantified,  and  evaluated  to  select  the  best  plan,  track  changes,  and  characterize  project  success. 

Metrics  used  in  plan  fonnulation  can  be  quantified  by  modeling  and  analysis,  or  can  be 
qualitative  in  nature,  while  metrics  used  to  assess  project  implementation  success  must  come 
from  direct  measurements  and  analysis.  Given  the  complexity  of  ecological  systems  and 
restoration  objectives,  a  multitude  of  potential  metrics  are  available  for  use.  With  limited 
funding,  it  may  only  be  possible  to  effectively  measure,  estimate,  or  otherwise  use  a  few  metrics, 
so  it  is  critical  to  carefully  select  the  metrics  that  can  clearly  show  the  system  state,  test 
hypotheses,  and  track  changes  in  ecosystem  quantity  and  quality  in  the  project  area.  Developing 
an  effective  yet  pragmatic  metric  set  necessitates  integrating  social  and  economic  considerations 
with  rigorous  scientific  concepts  of  ecological  structure  and  function,  while  effectively 
accounting  for  tradeoffs  among  measured  outputs  (Walters  1997,  Kiker  et  al.  2005). 

Many  current  methods  for  selecting  and  comparing  metrics  result  in  metric  sets  that  fail  to  capture 
the  information  necessary  to  develop  project  objectives  and  to  determine  the  degree  to  which 
management  actions  have  influenced  the  system  and  met  project  objectives.  The  lack  of  a  robust 
framework  for  selecting  a  set  of  restoration  metrics  may  result  in  sub-optimal  choices  that  are  ad- 
hoc,  mimic  historical  practices  and  regulations,  or  apply  screening  criteria  inconsistently,  leading 
to  restoration  planning  that  carries  subjective  bias  or  applies  insufficient  physical,  chemical,  and 
ecological  principles  based  on  outdated  assumptions  (Kondolf  1995,  Kondolf  and  Micheli  1995, 
Allen  et  al.  2002,  Nienhuis  et  al.  2002,  Neimeijer  and  de  Groot  2008). 

This  technical  note  is  intended  to  guide  restoration  planners  through  the  difficult  process  of 
selecting  effective  ecosystem  restoration  metric  sets  in  the  face  of  limited  resources,  varied 
project  objectives  and  stakeholder  priorities,  and  a  multitude  of  potential  metrics  to  choose  from. 
The  remainder  of  this  technical  note  reviews  methods  for  metrics  selection,  assessment,  and 
application  that  may  be  used  under  a  variety  of  circumstances. 
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REVIEW  OF  METRICS  SELECTION  METHODS:  Metrics  must  be  measurable  and  should 
have  desired  targets  or  outcomes  that  relate  to  project  objectives.  These  targets,  which  are 
specified  by  predictive  models,  data  mining,  or  by  best  professional  judgment,  are  used  to 
compare  predicted  project  plan  performance  to  actual  perfonnance.  To  address  the  challenge  of 
selecting  an  informative  and  appropriate  set  of  metrics  for  restoration  projects,  this  section  first 
reviews  and  assesses  existing  methods  of  identifying  possible  metrics.  Once  the  initial  set  of 
potential  project-specific  metrics  has  been  developed,  it  must  be  narrowed  to  the  optimal  metric 
set  depending  on  resource  constraints  and  stakeholder  priorities.  This  section  later  presents  two 
methods,  screening  and  multi-criteria  decision  analysis  (MOD A),  for  use  in  narrowing  metrics 
and  compiling  the  optimal  metric  set.  Metric  identification  methods  include  the  following: 

Conceptual  modeling.  Conceptual  models  generally  illustrate  relationships  among  key 
ecological  parameters,  target  species  and  performance  indicators,  and  attempt  to  simulate 
ecosystem  dynamics  and  describe  how  system  inputs  translate  to  outputs  (Harrington  and  Feather 
1996,  Ogden  et  al.  2005).  They  provide  a  link  between  early  planning  and  later  evaluation, 
synthesize  current  knowledge,  and  can  be  used  to  identify  appropriate  monitoring  metrics 
(Harrington  and  Feather  1996,  Thom  and  Wellman  1996,  USACE  2005,  USAGE  Environmental 
Advisory  Board  (EAB)  2006,  Fischenich  2008).  Ecosystem  models  can  also  be  expanded  to 
system  models  including  the  relevant  economic,  social,  political,  and  other  factors  affected  by  or 
influencing  enviromnental  restoration  projects.  Conceptual  modeling  is  an  iterative  process  that 
involves  multiple  revisions  and  expansions  to  create  models  of  varying  detail  that  describe  the 
relationship  between  relevant  (i.e.  those  influencing  the  project  objectives)  system  components. 
There  are  numerous  conceptual  model  types  that  differ  in  both  framework  and  presentation 
formats.  Common  conceptual  model  types,  described  in  further  detail  in  Fischenich  (2008),  include 
stress-response  models,  state  and  transition  models,  and  driver-stressor-endpoint  models. 

For  USACE  ecosystem  restoration  projects,  once  the  need  for  a  conceptual  model  to  identify 
appropriate  metrics  is  recognized,  proponents  and  appropriate  Planning  Centers  of  Expertise 
(PCXs)  may  use  the  conceptual  model  to  investigate  potential  desires  or  needs  for  planning  models 
that  might  be  used  during  a  study  to  evaluate  the  impacts  of  planning  alternatives.  Planning  models 
must  be  certified  and  approved  for  use  in  accordance  with  EC  1105-2-412.  In  addition,  conceptual 
models  that  illustrate  relationships  between  structures,  functions,  and  metrics  that  may  be  used  to 
quantify  each  have  been  observed  to  facilitate  communication  of  such  relationships. 

Though  a  number  of  approaches  exist,  the  general  conceptual  modeling  process  includes  seven 
steps  (adapted  from  Grant  et  al.  (1997)): 

1.  Formulate  model  objectives  (Why  is  the  team  creating  the  model?  What  do  they  hope  to 
accomplish/get  out  of  it?) 

2.  Determine  the  extent  of  the  model  (What  are  the  bounds  of  the  system?  Does  the  model 
include  natural  processes  only  or  natural  processes  and  socio-political  influences?) 

3.  Describe  the  boundary  and  other  assumptions  adopted  to  simplify  characterization  of  the 
system,  seeking  to  understand  the  conditions  under  which  the  assumptions  might  be  proven 
false  and  how  that  might  affect  the  validity  of  information  gleaned  from  the  model's 
application. 

4.  Identify  system  components  affected  by  or  influencing  project  objectives. 
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5.  Categorize  related  components  according  to  their  specific  roles  in  the  system  or  the  model 
and  establish  relationships  between  system  components. 

6.  Represent  the  conceptual  model  using  a  flow  chart  or  other  diagram  type. 

7.  Describe  expected  pattern  of  model  component  behavior;  usually  this  means  describing 
changes  in  model  components  over  time  in  response  to  specific  scenarios. 

8.  Review,  revise,  and  expand  the  model. 

Once  the  conceptual  model  has  been  created,  it  can  be  used  to  identify  and  form  a  comprehensive 
set  of  metrics.  From  that  set,  planners  should  seek  metrics  that  measure  the  most  important  system 
components  (those  impacting  the  project  objectives  or  ecosystem  quality  and  quantity)  identified  in 
the  conceptual  model. 

For  example,  in  the  USACE  guide  to  planning  aquatic  ecosystem  restoration  monitoring  programs 
(Thom  and  Wellman  1996),  the  authors  suggest  a  conceptual-model-based  approach  to  selecting 
metrics  that  links  project  objectives  to  perfonnance  parameters  (metrics).  The  project  team  creates 
an  ecosystem  model  that  relates  three  system  components:  controlling  factors  (e.g.  elevation, 
hydrology,  substrate),  structure  (e.g.  salt  marsh),  and  functional  components  or  resources  (e.g. 
waterfowl,  shorebirds,  fish).  The  model  is  designed  to  elucidate  key  factors  that  control  both  the 
development  and  maintenance  of  system  structure  and  attributes  and  functions  related  to  project 
objectives.  The  team  then  chooses  metrics  that  measure  aspects  of  each  of  these  components  while 
aiming  to  develop  a  scientifically  based,  easy-to-measure  metric  set  that  provides  useful  data  about 
the  system. 

The  approach  involves  three  steps  (adapted  from  Thom  and  Wellman  1996): 

1 .  Reduce  the  system  into  related  controlling,  structural,  and  functional  components  or  resources 
relative  to  project  objectives  (i.e.  create  the  conceptual  model). 

2.  Identify  measurable  metrics  within  each  component. 

3.  Compile  a  subset  of  metrics  from  the  initial  set  that  represents  all  model  components,  and 
measures  environmental  benefits  of  the  project  and  progress  toward  project  objectives.  At 
least  one  component  for  each  of  the  earth  science  “spheres”  that  can  describe  an  ecosystem 
should  be  selected,  including  physical,  hydrological,  and  ecological  measures  (National 
Resource  Council  (NRC)  1992). 

Another  example  of  a  conceptual  ecological  model  (CEM),  recently  developed  for  a  feasibility- 
level  monitoring  and  adaptive  management  plan,  is  based  on  a  top-down  hierarchy  of  infonnation 
organized  into  drivers,  stressors,  effects,  attributes,  and  perfonnance  measures.  Members  of  an 
interagency  project  team  created  the  model  by  reviewing  existing  information  about  the  ecosystem, 
meeting  to  discuss  reasons  for  natural  and  anthropogenically  driven  area  alterations,  and 
developing  lists  of  stressors  and  consequent  ecological  effects  (USACE  2010).  To  develop 
monitoring  metrics,  the  team  then  identified  attributes  that  could  serve  as  key  indicators  of  project 
success  through  assessment  of  related  performance  measures.  They  used  their  hypotheses  and  lists 
to  create  the  initial  CEM  draft,  which  was  then  further  revised  (Figure  1). 

Pros:  The  conceptual  model  method  can  provide  decision  makers  with  a  clear  view  of  important 
factors  and  their  relationships,  making  it  easier  to  develop  and/or  approve  of  a  metric  set  that 
satisfies  multiple  appropriate  lifecycle  needs.  These  relationships  reveal  what  should  be  considered 
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during  planning,  implementation,  and  monitoring  activities,  and  therefore  which  attributes  project 
metrics  should  aim  to  assess.  A  CEM  is  useful  because  it  requires  planners  to  contemplate 
connections  between  relevant  system  components,  and  helps  to  identify  key  project  components  to 
improve  ecosystem  structure  and  function  (Thom  and  Wellman  1996,  Gentile  et  al.  2001).  USACE 
policy  requires  the  use  of  models  for  all  planning  activities  and  USACE  guidelines  recommend 
conceptual  modeling  specifically  for  metrics  selection  (Thom  and  Wellman  1996,  EC  1 105-2-407 
(USACE  2005)).  The  use  of  a  CEM  also  provides  stakeholders  with  accountability  by  illustrating 
how  the  chosen  metrics  fit  into  the  system  and  address  specific  objectives.  In  addition,  it  can 
involve  and  incorporate  knowledge  and  opinions  from  a  variety  of  stakeholders  and  professionals. 
Early  formulation  of  a  conceptual  model  can  serve  other  potential  needs  as  well,  such  as  those  that 
might  be  associated  with  various  agency/USACE-specific  policies.  For  instance,  early  formulation 
of  a  CEM  can  provide  a  platform  to  initiate  discussion  of  planning,  review,  and  monitoring-related 
evaluation,  modeling,  and  monitoring  needs  that  can,  in  turn,  reveal  insights  about  the  potential 
linked  roles  to  be  served  by  metrics.  Depending  on  the  state  of  knowledge  in  a  particular  system, 
costs  and  time  can  range  from  relatively  low  to  moderate  compared  to  other  methods. 


Canal  Conatruction 


mpoundmant 


/  Decreased  \ 

x.  Productivity  y/ 


Land  Cover 

Vegetation 

Monitoring 

Elevation 

Water  Quality 

•  Spatial  Analysis 

•Percent  Cover 

•SET 

•TSS 

•  DBH 

•  Litteifall 


•  Feldspar 


•DO 

•  Nutrients 

•  Salinity 


Figure  1 .  Example  CEM  created  for  a  Louisiana  Coastal  Area  ecosystem  restoration 
monitoring  and  adaptive  management  plan  designed  to  restore  hydrologic 
connectivity  and  improve  fish  and  wildlife  habitat.  Top  rectangles  represent 
drivers,  ovals  represent  ecological  effects,  diamonds  represent  ecological 
stressors,  and  lower  rectangles  represent  attributes.  Bulleted  items  at  the  base 
of  the  diagram  are  performance  measures  or  metrics  that  correspond  to  each 
attribute  (from  USACE  (2010)). 


Cons:  The  conceptual  modeling  approach  to  selecting  metrics  often  does  not  assign  weights  or 
prioritize  the  model  components.  Therefore,  this  approach  does  not  aid  in  making  trade-offs 
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between  metrics  measuring  the  same  or  different  components  (USAGE  2010).  The  conceptual 
modeling  approach  still  leaves  room  for  bias,  as  best  professional  judgment  is  often  relied  on  to 
develop  the  model,  though  significant  documentation  and  involvement  of  stakeholders  or  a  wide 
variety  of  professionals  can  lessen  this.  Conceptual  models  are  simplifications  that  usually  focus 
only  on  the  ecosystem  components  deemed  most  relevant  while  leaving  out  less  important  or  less 
understood  factors. 

When  to  use  conceptual  models:  Conceptual  models  required  for  project  planning  should 
always  be  consulted  when  selecting  metrics.  The  approach  is  most  useful  when  knowledge  about 
the  specific  system  and  component  relationships  is  available  and  well  understood,  and  when  the 
most  important  factors  are  known.  Conceptual  modeling  can  be  used  in  a  situation  where  there  is 
little  funding  for  monitoring  and  evaluation  planning,  and  when  planning  needs  to  be  done 
quickly  if  a  suitable  model  already  exists  or  the  system  is  not  overly  complex.  Metrics  selection 
using  this  method  should  be  supported  by  other  methods  to  conduct  trade-offs  among  potential 
metrics  identified  with  the  CEM. 

Historical  precedence.  In  the  absence  of  more  structured  methods,  historical  precedence  (i.e. 
historical  practice)  is  often  used  to  infonn  metrics  selection.  In  this  method,  decision  makers 
review  metrics  used  in  similar  ecosystem  restoration  monitoring  and  evaluation  programs,  compile 
a  list  of  these  previous  metrics,  and  assess  and  narrow  them  down  based  on  specific  project  criteria 
and/or  screening  criteria  (see  “Screening”  below).  The  approach  can  be  simplified  into  three  steps: 

1.  Conduct  a  literature  search  to  identify  metrics  used  to  evaluate  similar  objectives  in  related 
projects.  The  literature  search  should  cover  projects  that  have  occurred  in  the  same  or  a  similar 
region  and/or  ecosystem  type,  respond  to  similar  disturbances,  involve  similar  stakeholders, 
and  have  similar  objectives  and  evaluation  criteria. 

2.  Compare,  evaluate,  and  refine  the  metric  list  using  screening  criteria  (e.g.,  is  the  metric  relevant 
to  the  ecosystem,  does  it  provide  useful  infonnation  about  the  ecosystem  state  and  specific 
project  objectives,  is  the  metric  sensitive  to  changes  in  the  system,  how  well  did  the  metric 
measure  success  in  previous  projects?) 

3.  Compile  a  subset  of  appropriate  metrics  from  the  initial  set  based  on  region  and  system 
specifics,  project  objectives,  and  ecological  concerns. 

Pros:  Maintaining  use  of  historical  metrics  often  allows  for  easy  comparison  to  baseline  data 
and  cross-comparison  among  projects,  particularly  within  a  region  or  restoration  program. 
Availability  of  similar,  previously  measured  data  makes  it  easier  to  set  measurement  goals,  and 
allows  managers  to  evaluate  system  conditions  compared  to  not  only  baseline  conditions  but 
historical  conditions  as  well.  Often,  metric  measurement  and  evaluation  methods  are  well 
established.  The  method  can  be  low  cost  and  time  efficient. 

Cons:  Choosing  metrics  based  on  historical  precedence  is  not  always  a  transparent  process  (i.e., 
reasons  for  selecting  the  metrics  are  not  always  clear  or  clearly  communicated),  may  hide 
individual  bias  (i.e.  certain  project-specific  objectives  may  be  implicitly  over-emphasized  or  de- 
emphasized  by  the  decision  maker),  is  less  rigorously  justifiable  than  other  methods,  and  may 
overlook  differences  in  context.  The  method  also  may  not  successfully  incorporate  new 
stakeholder  values  or  require  decision  makers  to  include  new  information  and  project  specifics  in 
the  decision-making  process.  Using  this  method  may  encourage  project  planners  to  choose 
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previously  used  metrics  that  may  not  be  most  suited  to  their  projects.  Historical  metrics  should 
be  used  with  caution,  as  multiple  modifications  may  have  caused  significant  changes  to  the  needs 
and  restoration  goals  of  the  system.  Additionally,  this  method  does  not  support  development  of 
new  measurement  techniques  or  technologies. 

When  to  use  historical  precedence:  This  method  should  be  used  when  restoration 
managers  are  able  to  review  past  projects  that  had  very  similar  ecosystem  characteristics  and 
project  goals,  and  in  which  practitioners  were  found  to  be  capable  of  evaluating  restoration  needs 
and  documenting  the  cause  of  observed  impacts.  Historical  precedence  should  be  considered 
when  the  previously  used  ecosystem  restoration  metrics  address  specifics  of  the  current  project, 
when  the  metrics  have  proven  to  be  useful  in  the  past,  and  when  the  use  of  metrics  can  be 
justified  with  respect  to  current  project  goals.  The  method  may  also  be  appropriate  for  low- 
budget  and  time-sensitive  projects. 

Best  professional  judgment  (BPJ).  In  this  method,  project  managers  or  larger  technical 
advisory  groups  select  metrics  directly  by  integrating  multiple  lines  of  science-based  and 
socioeconomic  reasoning  to  form  a  conclusion,  invoking  a  professional  opinion  that  is  case- 
specific  (Linkov  et  al.  2009).  Experts  may  be  interviewed  to  identify  potential  ecosystem  restora¬ 
tion  metrics  corresponding  to  decision  criteria  and  project  objectives  based  on  their  knowledge  of 
the  species  or  ecosystem  process.  Multiple  project  objectives,  relative  importance,  and  relation  to 
ecosystem  structure,  function,  and  benefits,  may  all  be  integrated  informally;  however,  significant 
documentation  to  justify  conclusions  and  recommendations  is  usually  required  (Jones  et  al.  2002, 
Oliver  2002,  Rohde  et  al.  2004,  Choy  et  al.  2009).  The  same  values  that  describe  desired  outcomes 
in  the  ecosystem  may  be  used  to  identify  success  criteria  for  assessing  actual  project  plan  per¬ 
formance  in  the  future.  In  other  cases,  the  success  criteria  may  relate  to  a  comparison  of  progress 
towards  the  idealized  project  outcome  with  the  current  baseline  and/or  predictions  of  future  trends. 

Pros:  BPJ  is  straightforward  and  generally  inexpensive  and  time-efficient.  It  emphasizes,  or 
sometimes  solely  involves,  the  judgment  of  individuals  who  are  experienced  and  familiar  with 
the  specific  case  of  ecological  restoration. 

Cons:  Metrics  selection  via  BPJ  may  exclude  or  place  bias  on  specific  stakeholder  values  or 
expert  experience  and  is  not  an  entirely  transparent  process  (i.e.  reasons  for  selecting  the  metrics 
are  not  always  clear).  It  is  challenging  to  correctly  integrate  a  wide  variety  of  influencing  factors 
(including  science-based  ecological  concepts  and  socioeconomic  concerns)  without  structured 
methods,  especially  when  dealing  with  complex  systems.  Without  assistance,  decision  makers 
may  have  difficulty  incorporating  and  evaluating  the  multitude  of  factors  and  relevant  details 
associated  with  metric  selection. 

When  to  use  BPJ:  BPJ  is  often  used  to  supplement  other  methods,  but  is  also  appropriate  for 
restoration  projects  involving  relatively  homogeneous  systems  (e.g.  a  small  wetland)  and/or 
projects  having  few  objectives.  The  method  is  also  more  appropriate  for  low-profile  projects  with 
few  stakeholders  and  little  public  attention.  In  other  scenarios,  it  may  be  necessary  to  use  BPJ  if 
the  project  is  limited  to  a  small  budget  and/or  more  rigorous  methods  are  not  feasible.  It  can  also 
be  used  for  complex  projects  once  more  structured  methods  have  simplified  the  problem. 
Additionally,  BPJ  can  be  used  to  narrow  options  for  metrics  to  an  optimal  list. 
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Narrowing  the  options:  Screening.  Restoration  project  managers  can  evaluate  or  “screen” 
potential  metrics  to  identify  the  most  appropriate  subset  of  metrics  for  a  given  project  (Table  1). 
Most  screening  criteria  are  applied  to  each  potential  metric  individually.  Each  metric  in  the 
selected  set  should  be  relevant,  unambiguous,  direct,  operational,  analytically  sound,  measurable, 
responsive,  and  anticipatory  to  the  greatest  extent  possible  (McKay  et  al.  2012).  The  remaining 
criterion  applies  to  the  entire  set — the  metrics  set  should  be  comprehensive.  Project  managers  can 
either  use  the  criteria  as  a  checklist,  or  rate  each  metric’s  potential  on  a  scale  for  each  of  the 
criteria,  if  higher  resolution  is  needed.  Screening  is  often  used  to  supplement  other  metric  selection 
methods. 
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Pros:  Screening  is  another  inexpensive  and  time-efficient  process.  Criteria  are  well-documented, 
and  project  managers  can  screen  potential  metrics  efficiently  without  specialized  programs. 
Screening  criteria  in  Table  1  satisfactorily  define  the  necessary  qualities  for  each  metric. 

Cons:  Screening  does  not  facilitate  fonnal  consideration  of  individual  metric  utility  within  the 
total  constellation  of  its  metrics  set  (Niemeijer  and  de  Groot  2008).  Most  of  the  criteria  apply  to 
individual  metrics,  and  there  is  no  internal  method  to  determine  whether  the  set  of  metrics  is 
comprehensive.  Additionally,  there  is  no  explicit  consideration  of  project-specific  stakeholder 
values  though  these  may  be  incorporated  using  additional  criteria. 

When  to  use  it:  Screening  is  a  valuable  step  in  metrics  selection  for  all  ecosystem  restoration 
projects,  if  one  or  more  of  the  evaluation  criteria  presented  in  Table  1  have  not  been  considered  in 
other  ways.  For  example,  since  MCDA  may  not  consider  whether  the  potential  metrics  are 
operational  or  measurable,  restoration  planners  can  use  screening  based  on  these  two  criteria  in 
conjunction  with  MCDA. 

Narrowing  the  Options:  Multi-criteria  decision  analysis  (MCDA) 

MCDA  modeling  is  a  structured  approach  to  decision  making  that  quantitatively  evaluates  project 
choices  (in  this  case,  metrics)  based  on  defined  criteria,  estimated  value,  and  stakeholder 
preferences.  MCDA  integrates  a  wide  variety  of  infonnation  to  evaluate  and  rank  potential  metrics 
based  on  their  ability  to  fulfill  designated  monitoring  plan  criteria  (e.g.  measuring  progress  and 
fulfillment  of  specific  project  objectives,  infonning  adaptive  management  planning,  etc.).  By 
providing  decision  makers  with  the  relative  utility  of  each  potential  metric  (both  overall  and  with 
respect  to  individual  monitoring  plan  criteria),  MCDA  analysis  allows  them  to  conduct  trade-offs 
among  the  various  choices  and  select  the  optimal  set. 

Selection  of  a  project-specific  set  of  metrics  is  tailored,  case  by  case,  from  a  general  set  of  metrics 
according  to  the  stakeholders’  preferences  and  the  set  of  criteria/subcriteria.  As  a  tool  for  metric 
selection,  a  Multi-Criteria  Decision  Analysis  is  performed  according  to  the  following  steps: 

1 .  Establish  a  set  of  metric  choices  to  be  assessed  and  ranked.  These  can  be  created  specifically 
for  the  project  at  hand  using  one  of  the  above  methods  or  come  from  an  existing  list  of 
metrics  (e.g.  Thayer  et  al.  2005).  See  McKay  et  al.  (2012),  also  from  the  EMRRP  technical 
notes  collection,  for  more  metric  development  guidance. 

2.  Formulate  comprehensive  criteria  and  subcriteria  that  represent  the  goals  of  the  monitoring 
plan  including  intended  benefits  and  important  aspects  of  the  restoration  project  objectives. 

3.  Elicit  preferences  from  managers,  planners,  and/or  stakeholders  to  establish  the  relative 
importance  of  the  criteria  and  subcriteria  and  assign  each  a  corresponding  weight. 

4.  Assign  each  metric  a  value  score  based  on  expert  assessment  of  the  metric’s  ability  to 
provide  useful  information  about  each  of  the  criteria  and  subcriteria. 

5.  Input  criteria,  subcriteria,  weights,  and  value  scores  into  the  MCDA  model.  (Other  inputs 
such  as  uncertainty  may  also  be  required,  depending  on  the  MCDA  method  used.  See  the 
input  section  below  for  a  more  detailed  description  of  each  input.) 

6.  Calculate  the  weighted  utility  score  and  ranking  of  each  potential  metric  that  represents  its 
contribution  to  the  overall  goal  of  the  monitoring  plan. 
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A  general  example  of  an  MCDA  model  for  metric  selection  (Figure  2)  includes  four  main  sections: 
overall  goal,  criteria,  subcriteria,  and  alternatives  (metrics). 


Figure  2.  MCDA  decision  tree  showing  example  overall  objective,  criteria,  subcriteria,  and  metric 
alternatives  for  a  sample  aquatic  restoration  project  and  how  they  are  related  and 
structured  within  MCDA  framework.  Here  the  overall  goal  is  restoration,  with  four  main 
project  criteria,  or  objective  categories,  including  ecology,  economy,  hydrology,  and 
geomorphology.  Main  criteria  then  have  subcriteria  that  describe  more  specific 
categories  of  objectives  (e.g.  hydrology  is  split  into  surface  and  groundwater  hydrology), 
so  that  each  of  these  objective  categories  may  be  weighted  and  scored  separately.  The 
right  side  of  the  model  shows  the  metric  choices  that  the  model  will  evaluate  with  respect 
to  the  criteria.  In  the  tree,  w,  represents  the  assigned  weights  of  the  criteria  and 
subcriteria  and  V(a,)  represents  the  value  score  of  each  metric  with  respect  to  each 
criterion  and  subcriterion.  The  utility,  and  therefore  the  ranking  of  each  metric,  a,,  is  a 
function  of  the  ability  of  each  metric  to  describe  the  criteria  and  subcriteria  (value  score) 
and  the  relative  importance  of  describing  those  criteria  and  subcritera  (weights). 


The  user  inputs: 

1.  Various  potential  project  metrics  based  on  established  lists  or  specified  by  the  project  team 
and/or  stakeholders. 

2.  Criteria  based  on  broad  project  objective  categories  defined  by  the  project  team,  technical 
advisory  team,  and/or  stakeholders. 

3.  Subcriteria  based  on  specific  project  objectives  within  the  main  criteria  and  defined  by  the 
project  team,  technical  advisory  team,  and/or  stakeholders. 
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4.  Weights  for  the  criteria  and  subcriteria  that  represent  the  importance  of  fulfilling  each 
criterion  and  subcriterion  relative  to  the  others  based  on  project  team,  technical  advisory 
team,  and/or  stakeholder  preferences. 

5.  Value  scores  that  represent  each  metric’s  ability  to  provide  information  about  each  criterion 
and  subcriterion  based  on  expert  evaluation. 

Once  users  have  input  this  infonnation  (and  additional  data  specific  to  the  MCDA  method)  into 
an  MCDA  software  program  and  run  the  model,  the  program  will  rank  each  metric  based  on  its 
contribution  to  the  overall  goal.  Model  inputs  can  easily  be  adjusted  to  detennine  the  effect  of 
varying  preferences  and,  with  certain  MCDA  methods,  varying  uncertainty.  Uncertainty  is 
considered  in  some  MCDA  methods  by  requiring  the  modeler  to  input  a  range  of  possible  values 
and  a  measure  of  the  certainty  of  those  input  values.  Instead  of  producing  a  single  rank  for  each 
metric,  it  calculates  the  probability  that  each  metric  will  have  each  rank.  Incorporating 
uncertainty  into  the  understanding  of  the  metrics  can  help  identify  project  alternative  plans  that 
are  more  robust  (i.e.,  capable  of  performing  over  a  wide  variety  of  uncertain  future  conditions) 
and/or  flexible  (i.e.,  able  to  adjust  in  the  adaptive  management  context  to  improve  probability  of 
achieving  ecosystem  restoration  goals  and  objectives).  Metrics  that  are  more  uncertain  may 
justify  the  need  for  a  more  robust  design  and/or  inclusion  of  contingency  options  in  the  project 
adaptive  management  plan.  For  more  infonnation  and  several  examples  of  MCDA  modeling,  see 
Linkov  and  Moberg  (2011). 

Pros:  MCDA  can  be  comprehensive  and  inclusive,  incorporating  modeling  results  and  expert 
opinion  on  a  variety  of  subjects  as  well  as  stakeholder  preferences  from  several  fields.  The  method 
allows  planners  to  simplify  complex  situations  with  varied  and  often  conflicting  options, 
objectives,  and  opinions.  New  and  changing  information  can  easily  be  integrated  and  assessed. 
Decision  makers  can  clearly  justify  management  choices  according  to  model  results  and  interested 
parties  can  easily  review  components  of  the  model  including  weights  and  alternative  scores.  The 
MCDA  method  thus  enables  restoration  project  managers  to  make  systematic  and  transparent 
decisions  (Reichert  et  al.  2007).  It  encourages  input  from  multiple  stakeholders  and  allows  decision 
makers  to  evaluate  the  effect  of  varying  preferences,  thereby  lowering  likelihood  of  biased  results. 
The  quantitative  results  allow  decision  makers  to  clearly  and  easily  compare  each  metric  choice. 
MCDA  standardizes  and  simplifies  complex  decision  making,  resulting  in  transparent,  robust, 
efficient,  and  accountable  restoration  planning. 

Cons:  MCDA  is  highly  dependent  on  the  comprehensiveness  of  the  criteria,  the  involvement  of 
stakeholders,  and  the  experts’  ability  to  accurately  quantify  each  metric’s  value  to  each  criteria. 
The  project  team  must  be  careful  to  choose  a  variety  of  stakeholders  to  avoid  bias,  and  must 
consult  knowledgeable  experts  to  provide  the  value  scores.  Done  well,  MCDA  can  be  time- 
consuming  and  can  be  more  expensive  than  other  methods.  Conversely,  when  done  poorly,  it  can 
complicate  communication,  and  obfuscate  the  planning  team’s  rationale  for  selecting  and 
prioritizing  metrics.  Also,  stakeholder  opinions  may  conflict  and  each  stakeholder’s  contribution  to 
the  assigned  weights  must  be  established.  Additionally,  the  method  is  designed  to  narrow  down 
chosen  metric  alternatives,  not  choose  them  explicitly,  and  does  not  include  guidance  for  choosing 
the  original  larger  metric  set. 
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When  to  use  MCDA:  This  method  should  be  used  when  many  diverse  stakeholders  are 
interested  in  the  project,  the  situation  is  complex  (several  objectives  and  metric  options  being 
considered),  and  several  months  are  available  for  conducting  the  MCDA.  It  can  be  useful  when  the 
situation  involves  multiple,  varied,  or  conflicting  objectives  and  competing  stakeholder  interests.  It 
is  also  useful  with  projects  involving  adaptive  management,  as  the  model  can  easily  be  updated 
and  the  system  reevaluated.  Restoration  managers  may  find  it  useful  to  use  screening  before  or 
after  MCDA  to  generate  an  initial  list  of  metrics  from  a  comprehensive  set. 

CHOOSING  A  METRICS  SELECTION  METHOD:  Metrics  selection  requires  careful 
consideration  of  a  multitude  of  factors,  but  above  all,  individual  metrics  must  evaluate  specific 
project  objectives  and  the  chosen  set  must  accurately  assess  overall  project  success  according  to  all 
relevant  criteria.  Choosing  metric  selection  methods  depends  on  a  number  of  factors,  including: 

•  project  scope 

•  degree  of  complexity  (e.g.  number  of  objectives  and  criteria) 

•  expert  knowledge  and  availability 

•  project  team  knowledge 

•  stakeholder  involvement  and  interest 

•  available  historic  data  and  similar  project  examples 

•  budget  and  time  constraints 

•  intended  use  of  information  gathered  from  the  metrics 

Investment  in  metric  selection,  in  terms  of  time,  money,  collaboration,  and  understanding  of  the 
system  will  likely  correlate  with  the  benefits  gained  from  monitoring  and  evaluation.  A  more 
appropriate,  robust  metric  selection  method  will  typically  lead  to  greater  project  transparency, 
defensibility,  and  utility.  In  general,  a  variety  of  metric  selection  methods  may  be  employed 
simultaneously  or  consecutively  to  reach  the  optimal  metric  set.  Additional  techniques  for  metric 
comparison  and  combination  can  be  found  in  McKay  et  al.  (2012). 

APPLYING  METRICS  TO  RESTORATION  EVALUATION  AND  MONITORING:  Metrics 
are  used  to  inform  planning  and  track  progress  toward  restoration  goals  throughout  a  project’s 
lifecycle.  During  the  project  planning  phase,  metrics  are  used  to  screen  management  measures,  and 
evaluate,  compare,  and  select  alternative  plans  (Engineer  Regulation  1105-2-100  (2-3)  (USACE 
2000)).  As  such,  it  is  important  to  consider  the  scalability  of  metrics  that  might  be  applied  during 
different  phases  of  a  project’s  lifecycle  (from  planning  through  adaptive  management  of  a 
constructed  project).  Metrics  applied  during  different  phases  should  be  selected  to  be  compatible 
and  complementary  with  one  another,  but  scaled  to  the  level  of  detail,  accuracy,  and  precision 
required  to  answer  specific  information  needs  of  the  study/project  phase.  Quantitative  metrics  used 
to  develop  evaluation  parameters  such  as  habitat  units  are  combined  with  qualitative  metrics  to 
describe  the  benefits  of  the  project  alternative  plan.  In  addition,  some  metrics  may  be  in  the  fonn  of 
measured  constraints  on  project  alternatives  (i.e.,  water  quality,  endangered  species,  water  supply, 
etc.)  and  are  also  incorporated  into  the  recommended  plan  selection  process. 

Once  a  plan  is  selected,  the  metrics  used  to  evaluate  potential  performance  should  also  be  used  to 
assess  actual  success  following  project  implementation.  If  uncertainty  is  associated  with  the 
achievement  of  the  project  objectives,  then  the  monitoring  parameters  and  corresponding  success 
criteria  should  include  additional  metrics  related  to  resolving  uncertainties  and  specific  triggers 
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or  thresholds  for  making  adjustments  to  improve  performance  based  on  the  project  adaptive 
management  plan.  Ecological  thresholds  are  often  characterized  as  a  point,  range,  or  distribution 
beyond  which  an  important  change  occurs  in  an  ecosystem  condition,  such  as  a  state,  pattern,  or 
process  (Bennetts  et  al.  2007).  The  subsequent  trigger  for  adaptive  management  action  should 
ideally  be  related  to  the  time  required  for  the  metric  to  have  responded  to  the  restoration  action 
based  on  values  of  the  monitoring  data  and/or  expected  recovery  time  from  the  literature. 

Information  gathered  from  the  chosen  set  of  monitoring  metrics  will  also  be  used  to  determine 
the  ultimate  effectiveness  of  restoration  actions  and  the  degree  of  overall  project  success.  This 
success  is  based  on  the  state  of  the  chosen  restoration  metrics  with  respect  to  success  criteria  that 
have  been  clearly  and  precisely  defined  during  pre-construction  planning  (CECW-PB  (USAGE 
2009)).  This  must  be  proved  using  the  pre-  and  post-project  implementation  values  of  the 
monitoring  metrics. 

Actual  project  results  are  compared  with  the  predicted  outcomes  of  restoration  activities  to 
indicate  the  degree  of  project  success.  First,  before  any  project  construction  or  ecosystem 
modification,  the  team  must  establish  baseline  values  of  the  chosen  metrics  under  pre¬ 
construction  conditions.  Often,  existing  data  can  be  used  to  establish  this  baseline.  Using  these 
baseline  values,  professional  judgment,  modeling  and/or  literature  review,  the  team  predicts  or 
sets  targets  for  post-construction  values  for  the  chosen  metrics  based  on  the  predicted  or  desired 
effects  of  their  restoration  actions.  Once  post-construction  monitoring  has  commenced,  predicted 
and  realized  metric  values  are  compared  to  measure  success. 

When  a  more  structured,  quantitative  method  of  evaluating  project  success  is  necessary,  such  as 
in  a  more  complex  project,  a  useful  method  of  determining  success  is  conducting  a  weight  of 
evidence  (WOE)  evaluation.  In  this  method,  threshold  metric  values  are  established  that  might  be 
project-specific  or  categorical  across  all  projects,  depending  on  the  metric.  Each  project  objective 
is  then  evaluated  as  “successful/unsuccessful”  (binary)  or  on  a  numerical  scale  for  each  metric. 
The  evaluation  based  on  each  metric  thus  generates  a  line  of  evidence  about  project  success  at 
achieving  stated  objectives,  and  can  be  used  to  document  and  prove  success,  or  advise  adaptive 
management  actions,  future  project  implementation,  or  restoration  program  priorities  (Linkov  et 
al.  2009). 

Planners  have  access  to  a  range  of  qualitative  and  quantitative  tools  to  integrate  multiple  lines  of 
evidence  into  a  cohesive  evaluation  of  project  success.  Choice  of  method  for  WOE  depends  on 
the  purpose  of  the  evaluation  (communication,  adaptive  management,  funding,  informing  future 
projects,  etc.).  Ordered  from  qualitative  (or  least  quantitative)  to  most  quantitative,  the  available 
methods  for  weight  of  evidence  evaluation  include:  listing  evidence,  best  professional  judgment, 
indexing,  and  quantitative  evaluation  (e.g.  MCDA)  (Linkov  et  al.  2009). 

Listing  evidence  is  the  simplest  application  of  WOE,  as  it  does  not  integrate  the  multiple  lines  of 
evidence  (Linkov  et  al.  2009).  If  an  assessment  of  overall  restoration  success  is  not  necessary, 
listing  evidence  may  be  appropriate  for  communication  to  specific  stakeholders  concerned  about 
individual  objectives,  informing  future  project  management,  or  guiding  adaptive  management. 
Other  WOE  methods  attempt  to  integrate  the  lines  of  evidence  to  form  a  conclusion  (Linkov  et  al. 
2009).  Best  professional  judgment  is  defined  as  qualitative  integration  of  multiple  lines  of 
evidence,  and  is  generally  case-specific.  Indexing  is  defined  as  integration  of  lines  of  evidence  into 
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a  single  measure  based  on  empirical  models.  Quantification  is  defined  as  integrated  assessment 
using  formal  decision  analysis  and  statistical  methods,  and  includes  MCDA  (Linkov  et  al.  2009). 
Quantification  is  the  most  transparent  and  therefore  often  most  useful  method  for  consensus 
building.  Specifically,  it  allows  scientific  lines  of  evidence  (here,  information  gathered  from  metric 
assessment)  and  social,  political,  logistical,  economic,  or  other  considerations  to  be  integrated 
explicitly  and  systematically  (Linkov  et  al.  2009).  Quantification,  especially  MCDA,  may  be  most 
appropriate  where  the  project  evaluation  based  on  multiple  metrics  must  be  used  in  further  decision 
making  such  as  adaptive  management  or  infonning  future  projects,  or  where  transparency  is 
important  for  communication  to  stakeholders.  Structured  and  quantitative  approaches  to  WOE  are 
preferred  where  possible  (Weed  2005,  McDonald  et  al.  2007,  Linkov  et  al.  2009). 

CONCLUSION:  This  technical  note  assists  ecosystem  restoration  project  planners  in  selecting 
the  best  metrics  to  identify  the  recommended  plan,  resolve  uncertainties,  and  monitor  and 
evaluate  progress  toward  project  objectives.  Choosing  metrics  that  effectively  measure  project 
objectives,  such  as  specific  environmental  benefits,  is  critical  to  effectively  plan,  monitor,  and 
evaluate  ecological  restoration  projects  and  make  informed  decisions  to  improve  project 
performance.  Project  managers  and  physical  scientists  often  lack  a  robust  framework  for 
identifying,  developing,  and  selecting  metrics.  This  technical  note  discussed  the  importance  and 
difficulty  of  choosing  appropriate  metrics,  and  reviewed  metric  selection  methods  including 
conceptual  modeling,  historical  precedence,  best  professional  judgment,  screening,  and  multi¬ 
criteria  decision  analysis.  A  number  of  factors  determine  the  best  metric  selection  method(s)  for 
each  project,  including:  specific  project  scope,  expert  knowledge  and  availability,  project  team 
knowledge,  stakeholder  involvement  and  interest,  available  historic  data  and  similar  project 
examples,  budget  and  time  constraints,  and  the  intended  use  of  information  gathered  from  the 
metrics.  This  document  highlights  the  strengths  and  weaknesses  of  each  method  and  suggests  the 
best  conditions  for  use.  Finally,  WOE  methods  were  examined  as  a  way  to  evaluate  restoration 
objectives  based  on  the  metric  data.  Thoughtful  and  deliberate  metrics  selection  and  utilization 
will  most  often  lead  to  increased  project  benefit  through  well-informed  planning  and  adaptive 
management,  greater  monitoring  and  evaluation  program  efficiency,  and  additional  scientific  and 
operational  knowledge. 

ADDITIONAL  INFORMATION:  Research  presented  in  this  technical  note  was  developed  under 
the  Environmental  Benefits  Analysis  (EBA)  Research  Program.  The  US  ACE  Proponent  for  the 
EBA  Program  is  Rennie  Sherman  and  the  Technical  Director  is  Dr.  Al  Cofrancesco. 

Technical  reviews  were  provided  by  Shawn  Komlos,  and  John  Wright. 

For  additional  information,  contact  the  corresponding  author,  Dr.  Igor  Linkov  (617-233-9869, 
Igor. Linkov(a), usace. army. mil),  or  the  manager  of  the  Environmental  Benefits  Analysis  Research 
Program,  Mr.  Glenn  Rhett  (601-634-3717,  Glenn. G.Rhett(d), usace. army. mil).  This  technical  note 
should  be  cited  as  follows: 

Convertino,  M.,  K.M.  Baker,  C.  Lu,  J.T.  Vogel,  K.  McKay,  and  I.  Linkov.  2013. 

Metric  selection  for  ecosystem  restoration.  EBA  Technical  Notes  Collection. 

ERDC  TN-EMRRP-EBA-1 9.  Vicksburg,  MS:  US  Army  Engineer  Research  and 

Development  Center.  htW://cw-environment.  usace. army. mil/eba/ 
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